Improve HTTP response handling and retry logic - #520
Merged
Merged
Conversation
- Add Authorization header with Basic auth (base64 encoded write key) - Add X-Retry-Count header on all requests (starts at 0) - Implement Retry-After header support (capped at 300s) - Retry-After attempts don't count against backoff retry budget - Add granular status code classification: - Retryable 4xx: 408, 410, 429, 460 - Non-retryable 4xx: 400, 401, 403, 404, 413, 422 - Retryable 5xx: all except 501, 505 - Non-retryable 5xx: 501, 505 - Replace backoff decorator with custom retry loop - Exponential backoff with jitter (0.5s base, 60s cap) - Clear OAuth token on 511 Network Authentication Required - 413 Payload Too Large is non-retryable - Add 30 new comprehensive tests (106 total tests) Aligns with analytics-java and analytics-next retry behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns with analytics-java change to accommodate shorter backoff periods (0.5s base, 60s cap). With faster retries, a higher retry limit allows for better resilience during extended outages. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR improves HTTP response handling and retry logic in the analytics-python client, aligning with changes from analytics-java and analytics-next. The implementation replaces the backoff decorator with a custom retry loop that provides fine-grained control over retry behavior, distinguishing between Retry-After attempts and exponential backoff attempts.
Changes:
- Replaced backoff decorator with custom retry loop implementing exponential backoff with jitter (0.5s base, 60s cap)
- Added Authorization header support (Basic auth with write key and OAuth Bearer token)
- Added X-Retry-Count header to track attempt numbers
- Implemented Retry-After header support (capped at 300s, doesn't count against retry budget)
- Granular status code classification distinguishing retryable from non-retryable errors
- Increased default max_retries from 10 to 1000 to accommodate faster retry cadence
- OAuth token clearing expanded to include 511 status code
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| segment/analytics/request.py | Added Authorization header (Basic/Bearer), X-Retry-Count header, parse_retry_after() function, and response object to APIError |
| segment/analytics/consumer.py | Replaced backoff decorator with custom retry loop implementing exponential backoff, Retry-After support, and granular status code classification |
| segment/analytics/client.py | Increased default max_retries from 10 to 1000 |
| segment/analytics/test/test_request.py | Added 17 comprehensive tests for authorization headers, X-Retry-Count, Retry-After parsing, and OAuth token clearing |
| segment/analytics/test/test_consumer.py | Added 13 comprehensive tests for retry logic, status code classification, backoff behavior, and Retry-After support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Extract duplicate exponential backoff calculation into helper function - Add upper bound (max_total_attempts) to prevent infinite retry loops with Retry-After - Improves code maintainability and prevents edge case of continuous Retry-After responses Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove 408/503 from Retry-After eligibility (only 429 uses Retry-After) - Add rate-limit state to Consumer (rate_limited_until, rate_limit_start_time) - 429 with Retry-After: set rate-limit state, raise to caller for requeue - 429 without Retry-After: counted backoff (not pipeline blocking) - Add maxTotalBackoffDuration / maxRateLimitDuration config (default 43200s) - upload() checks rate-limit state before request(), enforces duration limit - 511 OAuth gating: only retry when OauthManager is configured - Add tests: T04, T17, T19, T20; update 429/408/503 behavior tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Handle Retry-After: 0 correctly by checking 'is not None' instead of truthiness. Prevent silent batch re-queue on 429 without Retry-After by gating the upload() re-queue path on rate_limited_until being set. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
segment/analytics/consumer.py:158
- When a 429 with Retry-After is received (lines 136-144), the batch items are re-queued using
queue.put(), but then in the finally block (lines 156-158), all items in the batch are marked astask_done(). This means the items that were re-queued are still marked as done, which could causequeue.join()to return prematurely before those re-queued items are actually processed. Consider either not callingtask_done()for re-queued items, or handling the re-queueing differently.
if e.status == 429 and self.rate_limited_until is not None:
# 429: rate-limit state already set by request(). Re-queue batch.
self.log.debug('429 received. Re-queuing batch and halting upload iteration.')
for item in batch:
try:
self.queue.put(item, block=False)
except Exception:
pass # Queue full, item lost
success = False
else:
self.log.error('error uploading: %s', e)
success = False
if self.on_error:
self.on_error(e, batch)
except Exception as e:
self.log.error('error uploading: %s', e)
success = False
if self.on_error:
self.on_error(e, batch)
finally:
# mark items as acknowledged from queue
for _ in batch:
self.queue.task_done()
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add KeyError to except clause when parsing JSON response to handle missing 'code' or 'message' keys - Add explanatory comment on pre-existing except-pass pattern Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
requests.Response.__bool__() returns False for non-2xx status codes. The checks `if e.response` and `if response` evaluated to False for 429 responses, so parse_retry_after() was never called and the SDK fell back to normal backoff instead of respecting Retry-After. Changed both checks to explicit `is not None` comparisons. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add on_error handler to capture delivery failures from the SDK. Reports success=false with the first error message when any batch fails (non-retryable error or retries exhausted). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align retry configuration with cross-library defaults. Base backoff (500ms) and max backoff (60s) already matched. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
First request (retry_count=0) no longer includes the header. Retries with retry_count > 0 continue to send X-Retry-Count: 1, 2, 3, etc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns with the analytics-next Node reference implementation which treats all 200-399 responses as successful delivery. Previously only exact 200 was treated as success, causing 201/204/3xx to be misreported as errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Resolve python via activated venv/devbox, fall back to python3 - Use $PYTHON -m pip instead of bare pip (fixes macOS/nix where pip is not on PATH) - Add devbox setup as recommended path in README, with venv fallback instructions
Contributor
Author
Deep Code Review —
|
| Gap | Severity |
|---|---|
max_total_backoff_duration=None — triggers TypeError in consumer |
Critical (matches issue #2 above) |
max_rate_limit_duration=None — triggers TypeError in consumer |
Critical (matches issue #2 above) |
| Queue full during 429 re-queue — items silently dropped | Important (matches issue #5) |
Retry-After: 0 tight-loop behavior |
Important (matches issue #4) |
Retry-After with HTTP-date format falls back to counted backoff |
Nice-to-have |
Multi-consumer (thread=2) rate-limit state is not shared |
Nice-to-have |
sync_mode has no retry logic — failures propagate directly |
Documentation gap |
retries=0 with a retryable (non-429) error |
Minor |
The happy-path and primary retry scenarios are very well covered. The new test suite is a significant improvement.
Positive Notes
- The manual retry loop replacing
backoff.expogives much better control over retry semantics. The dual-budget model (per-attempt count + wall-clock duration) is solid. - Separating 429+Retry-After (pipeline blocking) from 429-without-Retry-After (counted backoff) is the right design.
X-Retry-Countheader omitted on first attempt is correct — servers can use presence of the header to detect retries.parse_retry_aftercapping at 300s is a good safety valve.- The
on_errorcallback hookup ine2e-cliis a real improvement for observability in e2e tests. - The Basic auth migration from
HTTPBasicAuthto explicitbase64is fine (equivalent behavior, removes one import).
- Remove unused backoff dependency from setup.py and requirements.txt - Reject None for max_total_backoff_duration/max_rate_limit_duration in client.py - Fix off-by-one: use >= for max_total_backoff_duration check - Fix Retry-After: 0 tight loop: fall through to counted backoff instead of pipeline-blocking - Log and call on_error when queue is full during 429 re-queue - Document flush() blocking behavior in docstring - Add comment explaining 410 retryable parity with Node SDK - Extract duplicate backoff logic into apply_backoff() helper - Warn on unrecognized Retry-After format (HTTP-date) instead of silently ignoring - Add comments for task_done() invariant and FatalError origin - Add 8 new tests covering all previously missing coverage gaps
Contributor
Author
Added 8 new tests covering gaps |
Removed unnecessary line in flush method documentation.
didiergarcia
previously approved these changes
May 11, 2026
Route any retryable response carrying a valid Retry-After header through the rate-limit path (no retry-budget cost) instead of special-casing 429. Retryable statuses without Retry-After continue to use counted exponential backoff. Adds 529 to the retryable set and covers both paths with tests. Matches the behaviour already shipped in analytics-java 3.5.5 and the generic-retry-after conformance suite in sdk-e2e-tests.
Resolves the conflicts introduced by #535 (ruff formatting at line-length 140, isort, and the setup.py -> pyproject.toml/uv migration). Resolution: keep this branch's retry semantics, then re-apply master's tooling standard. Master's changes to the conflicted files were verified to be formatting-only apart from import ordering, one unused-import removal and an E713 rewrite, so no behaviour from master was dropped. Also: - Drop the `backoff` dependency from pyproject.toml and uv.lock. This branch replaced the library with a hand-rolled backoff in consumer.py, and setup.py had already dropped it; the lock was still pinning a phantom dependency. - Fix two racy queue-full tests. They built a Client with the default send=True, so the consumer thread could drain the 1-slot queue between the two track() calls, failing roughly one run in four. Now they call client.join() first, matching the existing test_overflow idiom. Verified: ruff check, ruff format --check, and the full pytest suite (129) pass.
upload() used "self.rate_limited_until is not None" as its test for whether a failure was a rate limit, but that field was only cleared on success or once max_rate_limit_duration elapsed — never when its wait simply passed. After any rate-limit episode, every subsequent APIError, including permanently non-retryable ones, therefore took the re-queue branch instead of the drop branch. A 429 with Retry-After followed by a steady 400 re-queued the same batch forever: no sleep, because the deadline was already in the past, so roughly 16 uploads a second, on_error never called, and queue.join() — and so flush(), shutdown() and the atexit hook — blocked until max_rate_limit_duration, 12 hours by default. request() now marks the exception it raises from the rate-limit path, and upload() classifies on that instead of on consumer state. The episode gate is rate_limit_start_time, and rate_limited_until is cleared once its wait has been served, so a spent deadline can no longer classify anything. Two further fixes: - The Retry-After wait was a single uninterruptible sleep of up to MAX_RETRY_AFTER_SECONDS (300s), so pause(), join() and the atexit hook blocked for its full duration. It now waits in one-second slices and checks self.running, matching the bounded-shutdown behaviour of the other SDKs. - parse_retry_after read a naive HTTP-date in the host's local timezone. parsedate_to_datetime returns a naive datetime for the RFC 5322 "-0000" offset that servers do emit, so on a UTC+14 host a date 120s in the future parsed as 14 hours in the past and returned None — silently discarding the server's instruction and falling through to backoff. Naive datetimes are now read as UTC. 131 unit tests and all 58 e2e tests pass.
Cut the before/after narration from the comments added with the Retry-After work; the diff carries that. What is left states the invariant a maintainer needs: that rate_limit_start_time marks the episode while rate_limited_until is only the current deadline, and that upload() classifies on the flag rather than consumer state.
The header assertion in sdk-e2e-tests is opt-in per SDK, since analytics-kotlin and analytics-swift do not send it yet. This SDK does, so it runs the check.
Every one of these SDKs treated a 3xx as a failure before this work, and the change to 200-399 came from the design doc's "Spec item 1: 2xx and 3xx are success". That line is wrong, and the doc is what needs correcting. Measured against a local server, with the same HTTP clients these SDKs use: 307/308 + Location -> followed as POST with the body, arrives as 200 301/302/303 + Loc. -> followed as GET with no body, arrives as 200 302 without Location-> surfaces raw as 302 300 Multiple Choices-> surfaces raw as 300 304 Not Modified -> surfaces raw as 304 So a raw 3xx only reaches the classifier when the client has already declined to follow it, meaning nothing was uploaded. The one redirect that genuinely works, 307/308, never produces a 3xx here at all — it produces 200 — so narrowing the bound cannot break it. Nothing was gained by the wider range; a 300, 304, or Location-less 302 from a proxy was being logged as a delivered batch and dropped with no error callback. The narrower bound also needs no new branches: a 3xx is neither 5xx nor in the retryable 4xx set, so it already falls through to the non-retryable path and reports a failure. TAPI does not emit 3xx and has no plans to. This matters because host is customer-configurable and proxies in front of it are common. requests declines the redirect, so this replaces a misleading "Unknown error: [302]" with a named redirect failure. Tests split: 2xx is success, 3xx raises APIError with code "redirect".
Three findings from review. The shutdown path added with the interruptible Retry-After wait returned before the try/finally that discharges queue.task_done(), so every item taken by queue.get() for that batch left an outstanding obligation and queue.join() — and therefore flush(), shutdown() and the atexit hook — never completed. The batch is still handed back; the get() obligations are now discharged alongside, and the re-queued copies carry their own. The backoff waits were still bare time.sleep() calls, so only the rate-limit wait honoured pause(). Both now go through _wait and re-raise if the consumer stopped, letting upload() drop the batch through its normal error path. Four tests recorded time.sleep to assert the backoff schedule; because _wait sleeps in slices they now record at the _wait boundary, which is the delay they were really asserting. Durations were measured with time.time(), so a clock adjustment could expire or extend the rate-limit and backoff budgets. All seven duration sites in consumer.py use time.monotonic(). request.py keeps wall-clock time, since it is comparing against an absolute HTTP-date. 132 tests and all 61 e2e tests pass.
Records the retry/Retry-After work and, for the SDKs where a header is newly on the wire, an upgrade note: customers whose proxies allowlist request headers had uploads rejected by the already-released analytics-next change, and the same trap applies here.
No SDK retries a 3xx: every one classifies it as non-retryable and reports a failed upload. The notes claimed it was retried, which is wrong, and would have sent anyone debugging a proxy redirect looking for retries that never happen. Also scopes python's 511 line to the OAuth case, which is the one place the spec does allow a 511 retry, and php's new budget options to the LibCurl consumer, since Socket ignores them.
Shutting down during a counted-backoff wait dropped the batch and reported it through on_error, while shutting down during a Retry-After wait handed it back. The asymmetry favoured the rare case: the counted path is what a 500, a timeout or a 429 without Retry-After takes. A batch interrupted there has not exhausted its retry budget or its duration budget, so the wait was interrupted, not the upload failed. ShutdownInterrupted separates the two so upload() can tell them apart. The new test fails without the fix (queue empty, on_error fired) rather than passing either way. set_rate_limit_state now takes the parsed delay instead of re-deriving it from the response. Parsing an HTTP-date Retry-After reads the wall clock and truncates to whole seconds, so two parses of one response can straddle a second boundary: the first sees 1 and opens the episode, the second sees 0 and leaves rate_limited_until unset. The next attempt then found an open episode with no deadline and skipped its wait entirely. Its docstring also still said "from a 429 response", from before Retry-After was honoured on every retryable status. 133 unit tests and the full 61-test e2e suite pass; ruff check and format are clean.
post() reads the error body as payload["code"], which raises TypeError — not KeyError — when the body is valid JSON but not an object. A list, string, number or null all subscript that way, so the exception escaped the handler, surfaced from post() as a generic error, and the consumer's network-error branch retried it. A non-retryable 4xx was therefore retried ten times whenever the body was not a JSON object. The new test covers all four shapes and fails without this change with "TypeError: list indices must be integers or slices, not str". 134 unit tests, ruff clean, 61-test e2e suite passes.
Two examples had their quotes changed from single to double. Nothing to do with retry handling, and markdown is not something ruff formats, so it was a stray edit. Removing it so the diff a reviewer reads is only the change it claims to be.
didiergarcia
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements improved HTTP response handling and retry logic to align with changes in analytics-java and analytics-next:
Authorizationheader (Basic auth with write key, OAuth Bearer token)X-Retry-Countheader on all requests (starts at 0)Retry-Afterheader support (capped at 300s, doesn't count against retry budget)backoffdecorator with custom retry loop for better controlStatus Code Handling
Retryable 4xx: 408, 410, 429, 460
Non-retryable 4xx: 400, 401, 403, 404, 413, 422, and all other 4xx
Retryable 5xx: All except 501, 505
Non-retryable 5xx: 501, 505
Retry-After Behavior
Test Coverage
Changes
Modified Files
segment/analytics/consumer.py: Custom retry loop, status code classification, Retry-After supportsegment/analytics/request.py: Authorization header, X-Retry-Count header, parse_retry_after()segment/analytics/client.py: Increased default max_retries to 1000segment/analytics/test/test_consumer.py: 13 new test casessegment/analytics/test/test_request.py: 17 new test casesAlignment
This implementation aligns with:
🤖 Generated with Claude Code